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ABSTRACT 

An exploratory study into the bias associated %«ith 
faculty self-reporting of time and effort is reported* The paper 
delves into the question of whether or not there is bias in faculty 
reports of time and effort and whether a pattern to any such bias 
might appear. The methodology for conducting the study involved 
gathering faculty estimates of time/effort using both questionnaire 
and diary instruments. A method for determining and quantifying bias 
was established and used in gathering data. Based upon the results of 
statistical treatments applied to the data, some observations and 
conclusions were made concerning the presence of bias in faculty 
time/effort reports. It was concluded that a diary is superior to a 
questionnaire for self -reporting of time/effort because no reliance 
need be placed on memory or on an individual's ability to estimate 
average time expenditure* (Author/KM) 
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A STUDY ON BIAS IN FACULTY REPORTS 
OF TIME AND EFFORT EXPENDITURE 



Patrick H. Sullivan, D.B.A. 

This paper is a report on an exploratory study into 
the bias associated with faculty self-reporting of time and 
effort. The paper delves into the question of whether or 
not there is bias in faculty reports of time and effort and 
whether there is a pattern to any such bias that might 
appear • 

Background 

In order to put this report into perspective, let us 
review the circumstances which led to its preparation. In 
January, 1971, this writer was directed to determine how 
faculty members at the Florida State University divide their 
time among specified categories of academic activity. The 
purpose of the project was twofold: First, it was to pro- 
vide information for impending legislative hearings and 
second, it was needed to produce baseline information for 
the up-coming program budget and six-year plan. 

A brief review of literature in the educational 
field revealed that although there were many studies on 
faculty time and effort, no substantive studies had been 
performed on the subject of survey methods for faculty 
time/effort reporting. With the absence of such research. 



an attempt was made to determine how other universities 
were obtaining information on faculty effort distrihution. 
Letters were sent to the directors of institutional research 
of the forty United States universities larger than Florida 
State University, those forty offering the largest number of 
doctoral degrees in academic year 1969-70,^ 

These letters asked for information on how each 
university vent about obtaining information on its faculty's 
division of effort. Of the forty schools polled, thirty 
replied. Of that number, 16 required a faculty question- 
naire, 5 obtained the information from department chairmen, 
and 9 made no attempt to obtain the information. 

The results of the poll indicated that a mail ques- 
tionnaire was the common method of obtaining faculty time/ 
effort information. In all cases the faculty members were 
given f islected information as to the reasons for the ques- 
tionnaire. Typically, this information suggested such rea- 
sons as Federal reporting requirements, budgetary studies, 
resource analyses, building a data base, or providing infor- 
mation for committees on promotion and tenure. 

In each instance, questionnaires were distributed, 
<;ompleted by respondents, and then returned either directly 
to the central office or via the department chairmen or 
deans. In no case did the university reveal that any attempt 
had been made to determine whether or not respondents were 

^Lloyd A. Garrison, ••Preparation of Junior College 
Instructors, •• Junior College Journa l, xil (November, 1941), 
135-41 . f 
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biased in their reporting of effort. Responses were col- 
lected and tabulated without challenge or modification by 
the university. 

The results of this poll of laiiversities led to a 
decision to utilize a questionnaire instrument to determine 
how the faculty at the Florida State University divided its 
effort among selected activities. Thisi writer then con- 
structed an instriament that requested faculty information 
required by the uriiversity. This instrument utilized the 
best concepts and approaches of each questionnaire received 
from the other universities. The resulting questionnaire 
was distributed directly to each faculty member having a 
half-time or greater appointment. 

The writer then conducted faculty briefings and 
institutional meetings campus-wide on how to complete the 
questionnaire. The reaction to the questionnaire itself, 
the faculty concepts of the reasons for it, their appre- 
hensions about the uses or the misuses of the data, and 
their view of the process as a threat impinging upon aca- 
demic freedom were revealed as real forces at work among 
the faculty. These factors were seen to be significant and 
unavoidable forces impinging upon the reporting process. 

Given these attitudes of fear and apprehension on 
the part of many faculty members, this researcher began to 
doubt whether it is possible to obtain accurate and unbiased 
information on faculty activities through use of a question- 
naire instrviment. In other words, it was felt that it 



would not be possible to accurately measure output of the 
faculty resotirce* 



Contributing thoughts 

In a paper presented to the American Council on 
Education's 1959 Conference on Faculty Workload, Dr. 
Theodore Caplow recognized tnat there are an infinite nxunber 
of ways to measure faculty load, but all of these pose prob- 
lems. The first problem, he felt, is that there will be 
cheating when faculty load is measured by an administrative 
office. That this is so is not disreputable. It is a mat- 
ter of general observation that whenever worker output is 
measured by employers there is sOTie misrepresentation unless 

extreme (and in this case, Caplow felt, inappropriate) care 
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is taken to verify the figures. 

In commenting on the deviltries of faculty load 

studies. Reeves and Russell made the following comment that 

provides an excellent summation of the situation. 

The evaluation of faculty load is an extremely dif- 
ficult problem. Teaching duties and other profes- 
sional duties vary tremendously from institution 
to institution and individual to individual within 
the given institution. In fact, the factors involved 
in determining total faculty load are so ntunerous 
and so varied as almost to preclude precise deter- 
mination by any mechanical method. No thoroughly 
scientific method of measuring faculty load is now 
available. Sxisting measures are unsatisfactory 
and incomplete. The answers ar? not yet in. Yet, 



^Theodore Caplow, ••The Dynamics of Faculty Load 
Studies," in Faculty Work Load , ed. by Kevin Bunnell (Wash- 
ington, D.C.: American Council on Education, 1960), p. 69. 



as a practical necessity, some method of measuring 
and adjusting faculty load—even though approxi- 
mate — must be employed 

While measurement of faculty output has been a con- 
sistent and continual problem in higher education for a 
number of years, there are no significant difficulties with 
measuring the output of processes which are easily quanti- 
fied. Assembly lines producing automobile parts, electric 
components, 6-packs of Coca Cola, or carton? of cigarettes 
present no conceptual meas urement challenges. Engineers 
have been ingenious in devising methods of measuring, weigh- 
ing, and otherwise assigning numerical values to the out- 
puts of such operational processes. 

On the other hand, wherever an output or operational 
result involves the human resovurce, measurement becomes more 
challenging. The Hawthorne experiments gave evidence of 
this phenomenon. Under varying operating conditions worker 
output increased regardless of thechanging physical environ- 
ments devised by the experimenters. It was determined that 
the presence of outsiders (the experimenters) was itself 
affecting the performance of the workers • Studies since 
then have determ: .ed that either the presence of observers 
or the knowledge that performance is being measured may have 
an impact on the amount and type of action taken by the per- 
son being measured. 

^F. W. Reeves and J. D. Russell, ••Instructional 
Loads," College Organization and Administration (Indian- 
apolis, Ind,: Board of Education, Disciples of Christ, 
1929), pp. 165-82. 



Hypotheses 

In developing the methodology to attack the problem 
of examining whether (and to what degree) there is bias in 
reporting time/effort through the use of questionnaires, the 
following questions (hypotheses) were proposed; 

1. IS there any significant bias in reporting 
faculty time/effort on questionnaires? 

2. IS there any significant difference in reporting 
b?=»s between academic ranks? 

3. IS there any significant difference in reporting 
bias between academic disciplines? 

Methodology 

With no existing studies to use as a baseline or 
point of departure, the research techniques described herein 
are exploratory in natvire. That is, they attempt to deter- 
mine the presence and pattern of occurrence of any bias in 
reporting on self-completion type questionnaires. 

An ideal approach to this kind of problem of deter- 
mining bias would be to compare measured responses to known 
values. Since the known values did not exist, other pro- 
cedures were needed. 

As an alternate methodology to the above, the fol- 
lowing was decided to be a satisfactory substitute. Truer 
values than those available through the questionnaire were 
used in place of the absolute known values. These truer 



values were obtained through use of another measuring instru- 
ment more accurate than the questionnaire. 

A careful review of the literature revealed that a 
diary instrument, when used under carefully controlled cir- 
cumstances, will provide more accurate information on time/ 
effort than will a questionnaire. 

It was therefore decided that the test instr?Jiment 
to be used to provide an alternate source of information on 
faculty time/effort would be a diary log sheet. Faculty 
estimates of time/effort were gathered using both question- 
naire and diary instruments. For selected activities, the 
algebraic difference between questionnaire and diary esti- 
mates was obtained. This difference represented a measure 
of the bias in the questionnaire re5:;ponses. 

Bias, a term used throughout this paper, is a syno- 
nym for noise, anomaly, delay, or difference. The word 
"bias" carries with it no connotation of either goodness or 
badness, it has no connection with statistical bias. It is 
the difference between the real amount of time/effort 
devoted to certain activities and the apparent amount of 
time/effort devoted to those activities as reported through 
the information channel. 

Thus, the algebraic difference between a sample 
respondent's diary results and the questionnaire's reported 
norm for that cell represented the difference between the 
individual's actual activity and the estimated activity for 
individuals of that cell. The assumption was made that this 
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difference represented a measure of bias and not the bias 
itself. Due to a lack of any previous developed technique 
for determining bias, this researcher felt justified in 
using the above measxare. Further, it was hoped that the 
exploratory nature of this research provided sufficient 
license for experimentation. 

In view of the foregoing a sample of faculty diaries 
was obtained in such a way that it could be used for two 
types of tests: First,, to determine whether the diary data 
were significantly different frcxn the questionnaire -data, 
and second, to determine if it might be possible to detect 
significant differences in bias between academic ranks or 
academic disciplines. 

As for the activities to be measured, the original 
questionnaire contained a number of candidates. Selecting 
the particular values to be studied from this list of can- 
didates required the development of a set of selection cri- 
teria. 

A first consideration was that the value (or parame- 
ter) should be clearly defined and not easily misunderstood. 
A second criterion for selection of parameters was that they 
should be common to all faculty. 

In order to meet the criteria of being easily 
understood and easily quantified, activities were defined 
in such a way as to be consistent with the faculty's per- 
ception of the task. Measuring activities iij the category 
••academic support^* would not be desirable since this 



represents a set of activities grouped for budgetary con- 
venience and not a set of tasks performed by a faculty 
member. Therefore, a professor might not find it easy to 
perceive, and hence be unabl^- to correctly quantify his 
••academic support" activities. 

It was therefore decided to measure instructional 
activities using both the questionnaire and the diary. The 
activities measured and compared were: classroom teaching 
activities, individual instruction activities, and academic 
advising. 

Forty-eight faculty members were selected using 
randOTi procedures within twelve pre-selected strata. These 
strata or cells were arranged to include faculty from all 
instructional departments. These departments were divided 
into four academic discipline groupings: 

1. Professional Schools 

2. Social Sciences 

3. Physical and Quantitative Sciences 

4. Arts and Letters 

Within each discipline grouping four faculty of each rank 
were selected. An application of statistical processes to 
the data gathered from these people allowed for the explora 
tion of bias and its differences between ranks and between 
disciplines. 

For each of the forty-eight faculty members par- 
ticipating in the study, diary data were obtained for no 
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less than fourteen days during the Spring Quarter of aca- 
demic year 1970-71 • For each respondent seven sample days 
were obtained during the first five weeks of the quarter 
and seven more sample days from the last five weeks. Each 
of the two seven-day periods or sample weeks for a respon- 
dent contained one Sunday, one Monday, etc., such that each 
day of the week was represented. 

As each diary sheet was received back from a par- 
tic ij.ant, it was checked and the data placed onto computer 
cards. When all diaries were returned and the data punched, 
the statistical comparisons began. The average ntmiber of 
hours per week expended in each category as reported on the 
diary was algebraically compared with the similar values 
reported on the questionnaire. The differences between the 
two, called bias here, were subsequently operated upon sta- 
tistically. 

Findings 

The first question to be answered by this research 
was whether there was any statistically significant bias 
present. A simple Student's 't" test was used to make this 
determination. The results >f this test showed bias to be 
present in all three categories of instructional activity, 
in all cases the questionnaire estimates exceeded the diary 
tabulations, thus indicating an upward bias in the estima- 
tion of time expenditure on the questionnaire. 
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In order to respond to the questions or hypotheses 
concerning the detection of differences in bias between 
ranks or disciplines, a two-way blocking design analysis of 
variance (ANOVA) model was used. In this model individual 
bias were considered to be the ex^ e^.^al units, aca- 
demic disciplines were the blocks, and academic ranks repre- 
sented the treatments. This procedure yielded a foux block 

[ by three treatment (or twelve cell) segmentation of the 

population. 

Use of this model revealed no differences in bias 
between the three academic ranks of professor, associate 
professor, and assistant professor for any of the three 
categories of activity. That is to say, that although bias 
factors existed for each rank, there were no statistically 
significant differences in bias between ranks. 

When testing for differences between disciplines, 
however, there were statistically significant differences 
found in the activity category called classroom instruction. 
The levels of bias were as shown in Figure 1. 

Conclusions 

The findings of this research lead one inexorably to 
certain conclusions. The first of these is that where a 
questionnaire instrument is used to obtain self -reported 
estimates of time and effort expenditure, th,%re is a dis- 
tinct possibility that bias will be found in the responses. 
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Fig. 1. — Comparisons of Levels of Bias 
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The second conclusion is that wherever accurate esti- 
mates of time/effort are needed through self -reporting 
instruments, a diary should be used in lieu of a question- 
naire. The results of this study demonstrate the virtues 
of the diary where accuracy of data is a prime consideration. 
Diaries require respondents to write down their activities 
as they occur. No reliance need be placed upon memory or 
iipon one's ability to estimate average time expenditure. 
Thus, a diary is an ex post documentation of events that 
have occurred, while a questionnaire typically requests 
estimates of past, present, or future events. 
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The third conclusion concerns the existence of bias 
among groups of respondents • This study has shovm that 
people differ in their tendency to bias questionnaire 
responses • These differences occur with both individuals 
and groups* The differences between individual biases were 
expected, whereas the differences in group bias were not« 
The large differences between the bias reported by Arts and 
Letters discipline and the other three disciplines is ample 
evidence that groups of respondents can collectively, and 
without apparent collusion, bias questionnaire inputs in the 
same direction and to the same degree* 

A fourth conclusion may be drawn concerning the com- 
position of bias« Bias, as used in this research, is the 
algebraic difference between diary values and questionnaire 
values* Given this mathematical definition, one must care- 
fully question the factors that make up the bias* The 
existence of both accurate and inaccurate estimates of time 
expenditiure on the questionnaire lead one to believe that 
there are two major components co bias* The first of these 
is the ability, or inability, of the individual to estimate 
time expenditure. The second component, independent of one's 
estimating ability, concerns the existence of a conscious 
effort to alter these estimates once they have been made* 

Implications 

This study should be of interest to administrators 
and managers who contemplate the use of self-completion 
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instrtunents to obtain time/effort expenditure data from 
employees. Although it is clear that bias can exist in 
data obtained through questionnaires, it should be equally 
clear that it can occur on diaries or any other self- 
reporting instrument • The nature of the diary technique is 
such, however, that it takes a more deliberate and consistent 
effort on the part of a respondent to bias a series of dia- 
ries. Thus, although one cannot expect to find bias com- 
pletely absent in diary-produced data, one can expect to 
find its presence minimized. 

Administrators should also be aware of the trade- 
off between the need for accurate information and the cost 
of obtaining it. A carefully controlled diary census of all 
faculty would provide the most reliable estimate of faculty 
activity that is possible using self-completion devices. 

This procedure is quite costly to carry out. The 
costs associated with planning for such a project; printing 
costs for thousands of diary sheets, the expense of enve- 
lopes for mailing, the cost of mailing (either in postage or 
increased work force for campus mail), the personnel needed 
for processing completed diaries and tracking down missing 
diaries, and finally, the computer costs for tabulation and 
analysis; sum up to a large dollar amount. 

At some small sacrifice in accuracy but at con- 
siderable saving in dollar expenditures, one might conduct 
a diary study involving a carefully selectee^ sample of 
faculty, rather than a census of faculty. At even less 
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cost and with lesser precision of results, one might con- 
sider using a questionnaire census (or sample) of faculty. 

The researcher believes that for most applications 
it is neither necessary nor desirable to incur the costs 
associated with obtaining precise estimates of employee 
time/effort expenditure. If one is willing to accept the 
bias likely to occur in questionnaire-gathered estimates 
of time/effort expenditure, then the precision versus cost 
decision should be an easy one to make. Quite simply, with 
decreasing requirements for accuracy, concomitant cost 
reductions can be realized. 

In closing I would like to share with you the thoughts 
of a pioneer in the field of institutional research, Hugh 
Stickler. The following comment from his 1959 classic paper 
on faculty load studies provides an excellent summary for 
this paper t 

From the various studies to which reference has been 
made in this paper . . . 'iotal workload cannot 
simply be described or easily measured. Literature 
to date is confusing, fragmentary, and inadequate. 
Findings are frequently conflicting and/or incon- 
clusive. Relatively little real progress has been 
made in developing adequate techniques for the 
measurement of total faculty loads'. . . . That was 
the attitude when Reeves and Russell described it 
in 1929; it is still the situation thirty years 
later in 1959. ^ 

And what of 1973? 



W. Hugh Stickler, ••Working Material and Biblio- 
c,^-aphy on Faculty Load,^' in Faculty Work Load , ed. by 
Kevin Bunnell, p. 84 citing M. E. Haggerty, The Evaluation 
of Higher Institutions (Chicago, 111.; University of Chicago 
Pr --s, 1937), pp. 144-56. 
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